Unification-based Multimodal Parsing
نویسنده
چکیده
In order to realize their full potential, multimodal systems need to support not just input from multiple modes, but also synchronized integration of modes. Johnston et al (1997) model this integration using a unification operation over typed feature structures. This is an effective solution for a broad class of systems, but limits multimodal utterances to combinations of a single spoken phrase with a single gesture. We show how the unification-based approach can be scaled up to provide a full multimodal grammar formalism. In conjunction with a multidimensional chart parser, this approach supports integration of multipleelements distributedacross the spatial, temporal, and acoustic dimensions of multimodal interaction. Integration strategies are stated in a high level unificationbased rule formalism supporting rapid prototypingand iterative development of multimodal systems.
منابع مشابه
Clavius: Bi-Directional Parsing for Generic Multimodal Interaction
We introduce a new multi-threaded parsing algorithm on unification grammars designed specifically for multimodal interaction and noisy environments. By lifting some traditional constraints, namely those related to the ordering of constituents, we overcome several difficulties of other systems in this domain. We also present several criteria used in this model to constrain the search process usi...
متن کاملFinite-state Multimodal Parsing and Understanding
Multimodal interfaces require effective parsing and understanding of utterances whose content is distributed across multiple input modes. Johnston 1998 presents an approach in which strategies for multimodal integration are stated declaratively using a unification-based grammar that is used by a multidimensional chart parser to compose inputs. This approach is highly expressive and supports a b...
متن کاملUsing HPSG to represent multi-modal grammar in multi-modal dialogue
In order to realize their full potential, multimodal systems need to support not just synchronized integration of multiple input modalities, but also a consistent easy-of-using interface to isolate integration strategies from application ad hoc manner. As the range of multi-modal utterances supported is extended, type of input modalities are increasing, utterances being supported from individua...
متن کاملDeterministic Shift-Reduce Parsing for Unification-Based Grammars by Using Default Unification
Many parsing techniques including parameter estimation assume the use of a packed parse forest for efficient and accurate parsing. However, they have several inherent problems deriving from the restriction of locality in the packed parse forest. Deterministic parsing is one of solutions that can achieve simple and fast parsing without the mechanisms of the packed parse forest by accurately choo...
متن کاملQuasi-Destructive Graph Unification
Graph unification is the most expensive part of unification-based grammar parsing. It often takes over 90% of the total parsing time of a sentence. We focus on two speed-up elements in the design of unification algorithms: 1) elimination of excessive copying by only copying successful unifications, 2) Finding unification failures as soon as possible. We have developed a scheme to attain these t...
متن کامل